SIFT: A simple algorithm for tracking elephant flows, and taking advantage of power laws

The past ten years have seen the discovery of a number of “power laws” in networking. When applied to network trafic, these laws imply the following 80-20 rule: 80% of the work is brought by 20% of the flows; in particular, 80% of Internet packets are generated by 20% of the flows. Heavy advantage could be taken of such a statistic if we could identify the packets of these dominant flows with minimal overhead. This motivates us to develop SIFT, a simple randomized algorithm for indentifying the packets of large flows. SIFT is based on the inspection paradox: A low-bias coin will quite likely choose a packet from a large flow while simultaneously missing the packets of small flows. We describe the basic algorithm and some variations. We then list some uses of SIFT that we are currently exploring, and focus on one particular use in this paper—a mechanism for allowing a router to differentially allocate link bandwidth and buffer space to large and small flows. We compare SIFT’s performance with the current practice in Internet routers via ns simulations. The advantages from using SIFT are a significant reduction in end-to-end flow delay, a sizeable reduction in the total buffer size and an increase in goodput. We comment on the implementability of SIFT and argue that it is feasible to deploy it in today’s Internet.